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ABSTRACT 


Certain  imaging  applications  can  exploit  features  beyond  the  spatial 
resolution  of  a  sensor  through  computational  super-resolution  techniques. 

One  particular  method,  super-sampling,  involves  assimilating  multiple 
frames  of  digital  video  into  composite  video  frames  of  higher  average 
sampling  density  than  the  original  sensor.  Under  certain  conditions  of 
scene  motion  and  sensor  characteristics,  the  effective  resolution  is 
significantly  enhanced. 

1 .  Preliminary  on  Focal  Plane  Array  Image  Sampling 

An  ideal  scene  function  VF  (.q  ,x2  ,t)  describes  with  limitless  detail  the  radiant  intensity  of  the  world 
relative  to  camera  coordinates.  An  ideal  lens  geometrically  projects  this  function  onto  the  focal  plane, 

Sgeometric(Xl’X2’t)OC'¥ 

where  [m, ,  m2  ]  are  the  magnification  factors.  In  practice,  lenses  exhibit  limited  spatial  fidelity  and  the 
projected  optical  image  is  sampled  by  array  of  finite  sized  detectors.  The  resulting  signal  is  degraded  due 
to  the  separable  effects  of  lens  and  pixel  to  generate  a  pseudo-image ,*  expressed  in  the  space  and  spatial 
frequency  domains  as 

S pseudo  (*i ,  *2  >  0  =  k pixel  (*1  >  *2  )  *  ^  psf  (*1  >  *2  )  *  * S geometrical  >  *2  >  0 
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The  spatial  extent  of  the  photo-detector  pixel  with  dimensions  [a, ,  an  ]  serves  as  a  "box  car"  integrator  of 
the  geometric  image  with  impulse  response 
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The  spatial  frequency  response  of  such  a  pixel  is  the  separable  function” 
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The  lens  point  spread  function  can  be  characterized  in  the  spatial  frequency  domain  as  a  monotonically 
decreasing  function. 

Assuming  a  distortion-less  optic,  the  Modulation  Transfer  Function  (MTF)  of  a  unit-sized  pixel  in  one 
direction  of  continuous  spatial  frequency  is  graphically  presented  in  Figure  1 . 


Figure  1  Continuous  Space  MTF  of  a  Unit  Sized  Pixel 

Integrating  and  reading  out  the  photo-detector  array  with  detector  pitches  \b{  J)1  ]  at  a  clock  rate  t 
digitizes  the  pseudo-image,  creating  a  discrete  representation 


clock 


S  digital*1!’  n2  -  n3  ]  =  Spseudo(nA  >*2b2  ’  H  2*  clock  ) 


The  Discrete  Space  Fourier  Transform  of  the  sampled  image  is  a  periodic  extension  of  the  Continuous 
Space  Fourier  Transform  of  the  pseudo-image  3 
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where  discrete  frequency  is  related  to  continuous  frequency  by 
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The  MTF  of  the  same  unit  sized  pixel  sampled  in  a  100%  fill  factor  lattice  is  graphically  presented  in 
Figure  2  for  one  direction  of  discreet  spatial  frequency.  Note  that  the  low  spatial  sampling  rate  of  a  FPA 
introduces  significant  alias  distortion  of  all  but  the  lowest  spatial  frequencies. 
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Figure  2  Discrete  Space  MTF  of  a  100%  Fill  Factor  FPA 

Conventionally,  spatial  resolution  is  significantly  improved  by  introducing  a  blurring  Nyquist  optic  to  pre¬ 
filter  all  spatial  frequencies  above  the  sampling  fold-over  frequencies  at  ±  T . 

Clever  alternatives  to  such  a  blurring  optic  is  to  dither  or  micro-scan  the  FPA  imaging  system  to 
synthetically  increase  the  sampling  rate.  The  MTF  that  results  from  a  6-fold  reduction  in  detector  pitches 
[bt  J)2  ]  is  graphically  presented  in  Figure  3  for  one  direction  of  discreet  spatial  frequency. 
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Figure  3  Discrete  Space  MTF  of  a  100%  Fill  Factor  FPA  subject  to  6-fold  dithering 

Note  that  although  the  pixel  blur  is  enhanced  over  the  domain  [-  X,+XL  a^as  distortion  is  significantly 
reduced,  permitting  restoration  by  an  image-sharpening  filter  that  was  inapplicable  to  the  low -resolution 
image. 


2.  Motion  Estimation 

All  forms  of  super-sampling  require  re-assembly  of  a  temporal  cube  of  digital  video  into  a  single  image 
with  a  higher  spatial  sampling  density4.  The  success  of  such  processing  hinges  on  accurate  knowledge  of 
the  FPA  motion  relative  to  the  object  scene.  Initial  approaches  that  incorporated  a  mechanically 
controlled  dither  held  hope  that  the  inter-image  motion  would  be  a  known  uniform  shift  determined  by  a 
commanded  position  change5.  Disappointingly,  empirical  results  show  that  a  known-shift  model  is 
insufficient  to  characterize  the  FPA  motion  when  the  sensor  platform  has  additional  vibration  or 
translational  motion6.  The  next  iteration  is  to  estimate  the  true  shift  by  any  member  of  a  well-developed 
class  of  image  displacement  estimation  algorithms7.  Even  still,  this  is  insufficient  to  characterize  the 
motion  for  most  image  sequences  acquired  from  moving  platforms 8.  As  previously  reported,  we  generated 
very  promising  results  by  applying  numerous  image  displacement  estimators  across  the  image  and  fitting 


the  resulting  motion  field  to  an  affine  displacement  model  characteristic  of  classical  perspective 
transformations .  Current  research  is  exploring  wavelet  based  parametric  models 9  to  account  for  higher 
order  terms  of  the  basic  perspective  transformation  to  account  for  simple  geometric  distortions  of  the 
sensor  optics. 


3.  Results 

We  extend  our  analysis  from  128x128  MWIR  data  of  the  NRL  Threat  Warning  Program10  to  640x480 
MWIR  data  of  the  previously  reported  SASSY  program1 We  were  provided  several  short  (30-50  frame) 
sequences  from  an  airborne  ingress  into  a  collection  of  targets.  For  compliance  with  RS-170  video,  a 
300x480  window  of  imagery  with  a  2:1  interlace  was  digitally  recorded.  Hence,  every  FPA  integration 
window  alternated  between  even  and  odd  subsets  of  the  480  available  rows  to  generate  a  new  300x240 
image. 

Because  of  the  very  limited  temporal  data,  we  could  not  compute  a  fully  meaningful  affine  optical  flow 
model  as  previously  demonstrated.  Instead  we  utilized  a  strictly  translational  displacement  model  with 
limited  success,  and  implemented  a  5-fold  increase  in  the  spatial  sampling  rate  of  the  video  sequence. 


Figure  4  128x128  window  of  Data  (Left)  and  subsequent  5-fold  Super  Sampling  (Right) 


Figure  5  Original  Data  (Upper  Left),  Interpolated  Imagery  (Upper  Right)  Super  Sampled  Imagery 
(Bottom) 


4.  Conclusion 

Resolution  enhancement  through  temporal  super  sampling  has  been  demonstrated  on  a  number  of 
recorded  digital  video  sequences.  Most  significantly ,  none  of  the  video  sequences  involved  controlled 
micro-dither  scanning,  and  no  consideration  was  given  for  the  subsequent  resolution  enhancement 
analysis  performed  years  after  data  acquisition.  This  underscores  the  fundamental  robustness  of  the  super 
sampling  approach,  and  suggests  an  immediate  deployment  in  a  prototype  real-time  system  without 
further  need  for  the  development  of  micro-dithering  hardware. 
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